The size of aerospace software, as measured in source lines of code (SLOC), has grown rapidly. Airbus and Boeing data show that SLOC have doubled every four years. The current generation of aircraft software exceeds 25 million SLOC (MSLOC). These systems must satisfy safety-critical, embedded, real-time, and security requirements. Consequently, they cost significantly more than general-purpose systems. Their design is more complex, due to quality attribute requirements, high connectivity among subsystems, and sensor dependencies--each of which affects all system development phases but especially design, integration, and verification and validation.

The System Architecture Virtual Integration (SAVI) initiative is a multiyear, multimillion dollar program that is developing the capability to virtually integrate systems before designs are implemented and tested on hardware. The purpose for SAVI is to develop a means of countering the costs of exponentially increasing complexity in modern aerospace software systems.

A key element of the virtual system integration approach of SAVI is an SAE International standard called the Architecture Analysis and Design Language (AADL); the SEI played a leading role in developing this standard and an open source tool set, which is proving its value in enabling early discovery of defects through analysis of AADL models of embedded software systems. Our analysis showed that the nominal cost reduction for a system that contains 27 MSLOC would be $2.391 billion (out of an estimated $9.176 billion), a 26.1 percent cost savings. The original study reported here had a follow-on study to validate and further refine the estimated cost savings. In this blog post, which is excerpted from a recently published technical report, ROI Analysis of the System Architecture Virtual Integration Initiative, I present the results of a return-on-investment (ROI) analysis to determine the net present value (NPV) of the investment in the SAVI approach.

The program is sponsored by the Aerospace Vehicle Systems Institute, a research center of the Texas Engineering Experiment Station, which is a member of the Texas A&M University System. Members of AVSI include Airbus, BAE Systems, Boeing, the Department of Defense, Embraer, the Federal Aviation Administration, General Electric, Goodrich, Aerospace, Hamilton Sundstrand, Honeywell International, Lockheed Martin, NASA, Rockwell Collins, and Sikorsky.

Growth in Size of Aerospace Software

Experts have long observed that software systems are growing in size, not only in the number of subsystems but also the degree of interactions between them. This condition has driven the exponential growth in software cost and likely will further raise the defect-removal cost, as each defect affects a larger number of subsystems, and will require innovative solutions that include incremental verification of functional and non-functional properties on software system architectures throughout the development life cycle.

Understanding the dynamics of defect introduction and removal the phases in which defects are introduced, the phases in which they are detected, and the cost of removing defects relative to the phase lag between introduction and detection is paramount to accurately estimating the rework cost in terms of total software cost. What's more, the dominance of rework cost resulting from requirements and architectural design defects suggests a strong need for improved techniques to prevent and detect such defects.

The main problem is clear: most defects are introduced in the early pre-coding phases of development, such as requirements and design, but the majority of defects are detected and removed in post-coding phases, such as integration and testing. As our report details, the nominal cost of removing a defect introduced in pre-coding phases and detected in post-coding phases is generally one order of magnitude higher than the cost of removing it prior to code development. For safety-critical systems, the difference can be as much as two orders of magnitude higher.

Summary of Methods

We analyzed the economic effects of the SAVI approach on the development of software-reliant systems for aircraft compared to existing development paradigms. The purpose of our return-on-investment (ROI) analysis was to determine the NPV of the investment in the SAVI approach. The investment into the approach over the multi-year SAVI initiative by the different member companies was estimated to be $86M. Their investment covers the maturation, adaptation, and piloting of SAVI practices and technologies, and the transition of the approach into member companies. We used conservative estimates of costs and benefits to establish a lower bound on the ROI; less conservative figures would demonstrate higher economic gains.

We determined the rework cost avoidance based on SAVI practice by applying an efficiency rate for removing defects to the rework cost of a system, compared to current practice. The approach included the following conservative assumptions:

• We adopted COnstructive COst MOdel (COCOMO) II, the leading tool for estimating software-development costs using the current development process. Using typical development processes, we derived the total cost for developing three software systems of different sizes as follows:
  • Each system consists of three types of subsystems safety critical, highly critical, and less critical with code bases of 30 percent, 30 percent, and 40 percent, respectively, of the total code base, a typical mix in aircraft industry. This let us differentiate the cost of subsystems with respect to their requirements.
  • Each subsystem is developed with both new code and reuse of existing code. We considered three cases of new code development and varied the proportions of code reuse from 30 percent to 70 percent.
  • We used three system sizes, two based on the current generation of aircraft software systems (27 and 30 MSLOC) and one reflecting a future system of 60 MSLOC. The synthetic system clearly illustrates the economic impact of system growth, although building a system of this size is unaffordable.
  • The nominal labor rate is $28,200 per month for 2014, based on 2006 data of $22,800 = $150/hr. * 152 hr./mo. * 1.02694*(2014 - 2006), adjusted for annual inflation at 2.694 percent.

• On the basis of SAVI members' experiences, we estimated the total system development cost from an estimate of the software-development cost using a multiplier of 1.55, which reflects software development making up about 66 percent of system development cost.
• On the basis of documented and experiential evidence for aerospace systems, we used two conservative estimates for total rework cost: 30 percent and 50 percent of the total system development cost.
• We determined ROI and NPV based on rework cost reduction attributed to earlier discovery of defects and did not include reductions in maintenance cost and deployment delays. We limited rework cost savings to discovery of requirements errors, which make up 35 percent of all errors and 79 percent of the rework cost.
• We used experts' estimates of the efficiency rate for removing defects of 66 percent as well as a reduced rate of 33 percent for more conservative estimates.
• We assumed that SAVI practices of model creation and analysis would replace existing document-based practices of system requirement and design specification at a similar cost.
• We used $86M as estimated investment by the SAVI member companies over multiple years to mature SAVI and transition current practice to SAVI.

Summary of Findings

Our analysis produced the following outcomes and observations:

• Approximately 70 percent of defects are requirements and design defects, but fewer than 10 percent are detected in these early phases. The rework cost of correcting defects introduced in requirements and design but detected late in the development lifecycle is one to two orders of magnitude higher than the cost of correcting them before coding. Requirement related re-work cost amounts to 79 percent and design-related rework cost to 16 percent of the total rework cost.
• In the most conservative scenario, for a 27 MSLOC system, the smallest cost avoidance is $717 million (out of an estimated $9.176 billion cost of development, a 7.8 percent cost savings). This is with 70 percent reuse, rework cost as 30 percent of total system development cost, and a removal efficiency of 33 percent. The arithmetic and logarithmic (continuously compounded) ROIs are 7.3 and 2.12, with an NPV of $263 million.
• The nominal cost reduction for a 27 MSLOC system is $2.391 billion (out of an estimated $9.176 billion, a 26.1 percent cost savings), occurring at 70 percent reuse, rework cost as 50 percent of total system development cost, and 66 percent removal efficiency. The arithmetic and logarithmic ROIs are 26.8 and 3.33, with an NPV of $1.076 billion.
• The cost reduction is linear to the rework cost and removal efficiency. With other factors held constant in a scenario, each unit increase in removal efficiency for requirements errors resulted in a cost reduction. For example, for the 27 MSLOC system with the highest degree of reuse, each 1 percent increase in removal efficiency resulted in a cost reduction of $22 million.
• Cost reduction for a given system size is also linear to the amount of reuse. However, comparing the cost reduction for systems of different size, we observed that the cost reduction for 60 MSLOC was more than twice that for 30 MSLOC. This is to be expected given the more than linear increase in interaction complexity in a larger system.

The predicted returns were considered to be higher than anticipated, which led to several follow-on activities. First, one of the SAVI system integrator members obtained an independent assessment by its organization's cost-estimating group that agreed well with the findings of this report. Second, this initial ROI study was followed by a second SAVI ROI study. In the second study, a Monte Carlo algorithm was used to drive the COCOMO II cost estimation, resulting in reduced variation of the results. In addition, the commercial tool SEER was used to build a SEER-SEM and SEER-H model of a Boeing 777-200 in order to explicitly estimate the cost of the non-software portion of the system and compare it to both publicly available data and estimates of the original SAVI ROI study presented here. The SEER analysis confirmed that the cost multiplier of 1.55 was acceptable for 2010. Unfortunately, the software count increases while the physical parts count remains stable, resulting in software increase from 66 percent in 2010 to 88 percent of the total system development cost by 2024.

The second study also considered tailoring the ROI analysis to reflect a subcontractor. This includes adjusting the scaling factor used to estimate the total cost relative to the software cost, the degree of software reuse, the overall investment in the SAVI technology, and personnel cost factors.

Implications for Software-Reliant Weapons Systems

Although the virtual system integration approach of SAVI has been advanced by the commercial aircraft industry it has also been piloted on several Department of Defense projects. It has been used to identify issues in existing embedded software systems, as well as proposed upgrades to existing systems. Most recently it has been applied by multiple contractor teams on the Joint Multi Role Technology Demonstrator effort in order to mature and transition this Architecture-Centric Virtual Integration Practice (ACVIP) for use in Future Vertical Lift. The virtual system integration approach has been used effectively to address cyber security issues in unmanned vehicles under the DARPA High-Assurance Cyber Military Systems (HACMS) program.

Additional Resources

Read the full report, ROI Analysis of the System Architecture Virtual Integration Initiative.

Read other blog posts about AADL and virtual integration.

Watch a webinar about Architecture Analysis with AADL.